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Abstract — We characterize the achievable range of perfor- 
mance measures in product-form networks where one or more 
system parameters can be freely set by a network operator. 
Given a product-form network and a set of configurable 
parameters, we identify which performance measures can 
be controlled and which target values can be attained. We 
also discuss an online optimization algorithm, which allows a 
network operator to set the system parameters so as to achieve 
target performance metrics. In some cases, the algorithm can 
be implemented in a distributed fashion, of which we give 
several examples. Finally, we give conditions that guarantee 
convergence of the algorithm, under the assumption that the 
target performance metrics are within the achievable range. 

I. Introduction 

Many stochastic systems are modelled using Markov pro- 
cesses. Models range from communication systems, com- 
puter networks and data center applications to content dis- 
semination systems and physical or social interactions pro- 
cesses [13], [17]. In particular, the framework of reversible 
Markov processes [12], [14] allows for an extensive analysis 
of such systems, often geared towards optimizing perfor- 
mance measures such as sojourn times, queue lengths and 
holding costs [2], [5], [23]. 

Instead of optimizing performance measures, we are in- 
terested in identifying which performance measures can be 
achieved. Any performance measure 7 for which there exist 
finite parameters r° pt such that the performance of the system 
equals 7, is called an achievable target. The collection 
of achievable targets is called the achievable region. A 
parameter is anything that changes a transition rate, such 
as processing speeds, number of servers and job sizes. 

In this paper, we describe which performance measures 
can be influenced in product-form networks [1], [11], [12], 
[14], given a list of configurable parameters. Our work makes 
explicit that the more configurable parameters one has, the 
more control one can exert on a system. We also identify 
the achievable region of these performance measures by 
assuming that parameters are unbounded, and that in this 
case the achievable region is a convex hull of a set of 
vectors. Which vectors there are in this set depends on which 
parameters there are, as well as which states there are in the 
state space il. 

Using our analysis of the achievable region, operators can 
know which performance measures the operator influences 
when changing parameters. By examining the achievable 
region of a system, an operator can furthermore know which 
performance measures are achievable. Supposing that an 



operator wants its system's performance measures to equal 
a certain achievable target 7, we then proceed to show how 
the (distributed) online algorithm in [22] can be used to find 
r opt . Related ideas on using online algorithms to optimize 
networks can be found in [19], [20], [21]. 

The online algorithm in [22] is a stochastic gradient 
algorithm [3], [6], [15], known to converge when operators 
can set parameters in a compact set and when operators 
choose appropriate step sizes and observation periods. We 
note however, that the conditions that guarantee convergence 
as described in [22] are insufficient when parameters are 
unbounded. The conditions need to be more stringent in 
order to avoid extreme parameter growth. We shall capitalize 
on the proof methodology presented in [22] in order to 
derive sufficient conditions to guarantee convergence with 
probability one for the application we have in mind here. 

In related work, Jiang and Walrand [9], [10] developed a 
distributed online algorithm that tells nodes in CSMA/CA 
networks (transmitter-receiver pairs) how to set their activa- 
tion rates so that their throughput equals a given target. They 
also identified the achievable region of the throughput under 
the assumption that activation rates can be unbounded. This 
paper generalizes and extends their results to the broad class 
of product-form networks. 

This paper discusses two topics and is organized as 
follows. The first topic is the achievable region. We describe 
our model in i lll- Al and identify the achievable region in ifll-BI 
for several examples in ifll-CI We also provide an in-depth 
discussion as to how we identified the achievable region in 
i UI-DI The second topic is finding parameters such that the 
performance measure of the system attains some target value 
from within the achievable region. We describe how to use 
an online algorithm to find these parameters in illll-AI and 
we provide sufficient conditions to guarantee convergence in 
EIIII-BI A proof that these conditions are sufficient is then 
given in 31TLC1 

II. Achievable region 

Throughout this paper, we denote by hi the i-th ele- 
ment of vector b. When taking a scalar function of an n- 
dimensional vector b, we do this element-wise, i.e. exp (b) = 
(expfri, ...,exp6„) T . If we have a |f2| -dimensional vector b 
in which each element corresponds to some state x £ fi, we 
write b x for that element of b that corresponds to state x. 
Similarly, we denote by Aij the element in row i, column 
j of matrix A. If rows and/or columns correspond to states 



in il, we write A XiV instead. Finally, we denote by 1„ the 
n-dimensional vector of which all elements equal one and 
by e„ i the n-dimensional vector with all of its element 
equalling zero except for a one in the i-th position. 

A. Model description 

Consider an irreducible, reversible Markov process 
{X(t)} t >o on a finite state space with generator matrix 
Q G Rl^TxIf 2 ! We consider cases where the Markov process 
models a system in which an operator can change one 
or more transition rates. We assume that an operator only 
changes transition rates in such a way that the process 
remains irreducible and reversible, and to avoid trivialities, 
we assume that there are d > configurable transition 
rates. We call the logarithm of such a configurable transition 
rate a parameter r^, where i = l,...,d, and collect all 
parameters in the vector r = (ri, rd) T . In other words 

for i = 1, d, there exist x, y 6 i7 such that r; = lnQ Xiy . 

Under these assumptions, the process has a steady-state 
distribution 7r(r) that can be written in the product form 



n(r) = 



1 



Z(r) 



exp (Ar + b), 



(1) 



ini xd 



is a matrix, r G 
t , 



b e 



where A G 

vectors and Z(r) = l|n| T exp (Ar + b) is the normalization 
constant. The matrix A tells us not only which, but also 
by how much parameters influence steady-state probabilities, 
while the vector b contains all kinds of other constants such 
as logarithms of rates that are no parameters of the systems. 

When operators change parameters, the steady-state prob- 
ability distribution changes. In particular, A T 7r(r) changes, 
because the elements 



(A T n(r)) . = ^2 A x,iK x (r), i = 1, ...,d, 



(2) 



are aggregates of steady-state probabilities. These aggregates 
typically have a physical interpretation. For example, if the 
service rates /i; of queues i = l,...,d in a closed Jackson 
network can be controlled and one defines = ln/z,, the 
right-hand side of (f2j) reduces to the (negative) mean number 
of customers in queue i. We come back to this and other 
examples later, and we then make the analysis explicit. 

B. Achievable aggregrate probabilities 

Given some vector 7 G R d , we are interested in finding 
finite parameter values r opt such that ^4 T 7r(r opt ) = 7. We 
call 7 our target. It is not a priori clear whether such values 
exist, but if they exist and if they are finite, we call the target 
achievable. In this paper, we identify a collection of target 
vectors that are achievable, which we call the achievable 
region A. 

Theorem 1. Any 7 G .4 = {A t q;|q; g (0, a T l| f2 | = 
1} is achievable. 

Furthermore, if B 66 = R™ xd , n < d, is an affine trans- 
formation, there exists finite r opt such that _B^4 T 7r(r opt ) = 7' 
for all 7' G {BA T a\a G (0, l)l n l, a T l|n| = l}. 



Note that A is the interior of the convex hull of all 
transposed row vectors of A. This can be seen by writing 



A T a = 



i—l xGQ 



(3) 



xen 



where A x ,. denotes the row vector in matrix A corresponding 
to state x. In the special case of modelling a CSMA/CA 
network, [9], [10] found that the achievable region of the 
throughput is the interior of the convex hull of all indepen- 
dent sets of an interference graph. We note that if we capture 
their system in our notation, A would have all independent 
sets of the interference graph as row vectors. 

Before we prove Theorem [1] which is the topic of iUI-DI 
we discuss several examples to which Theorem Q] can be 
applied. 

C. Examples 

1 ) Finite-state birth-and-death process: Consider a birth- 
and-death process on = {0, 1, 2, n). When the system 
is in state x G {0, 1, ...,n — 1}, it goes to state x + 1 after an 
exponentially distributed time with mean l/X x+ i. Similarly, 
when the system is in state x G {1, ...,n}, it goes to state 
x—1 after an exponentially distributed time with mean l/v x . 
The steady-state probability of observing the system in state 
x G VL is given by n x = Z" 1 f7j =1 XJ^. 

Suppose that we can change for i = 1, n — 1 in this 
system. With = In we know that 



Z(r) 



X X 

exp (E n ~ E ln 



p,i 



(4) 



which corresponds to ([]]) for A x ,i — t[x > i] and b x = 

For i = 1, ...,n, the right-hand side of (O expresses the 
steady-state probability P[X t > i] and using Theorem Q] we 
can characterize its achievable region. While the achievable 
region of V[X t > i] and control thereof is interesting, we 
want to instead determine the achievable region of each 
steady-state probability P[X t — i]. For this, we define a 
matrix B G M. nxn element- wise by setting B r c = l[r = 
c] — l[r = c + 1] for r,c = l,...,n. We can then use 
Theorem Q] to conclude that for any 7 G (0, l) n such that 

E"=i T» = 1» there exists A e (°! °°) n so that ¥ l x t =i] = 
7i for i = 1, n. Note that in this example, one has enough 

(and appropriate) parameters to control the entire steady-state 

distribution. 

2) Closed Jackson network: Consider a closed Jackson 
network with d G N queues and n G N permanent customers. 
A customer that leaves queue i G {l,...,d}, joins queue 
j G {l,...,d} with probability Pij. Customers are served 
at queue i with rate u^. The state space is given by fl = 
{y G N d \ Yli=i Vi — n }> w i m x i ^e number of customers 
in queue i. The steady-state probability of observing state 



x G Q is tt x = ^ 1 Ili=i( A iM) x< - Here ' A 6 M d is a 
non-zero solution of A = AP. 

Suppose now that we can change fa for i — 1, d. Define 
Ti = In /ii for i = 1, d so that 

_^ d d 

= ZfrT 6XP {~^2 Xin + ^l Xi lnA *)> ( 5 ) 

^ ' i=l i=l 

which is equivalent to (Q]) when A x i = —Xi and b x — 

After substituting A^.j = — xi into (0 we see that the 
right-hand side of (f2]l can be interpreted as the negative 
steady-state expectation of the number of customers in queue 
i. Using Theorem Q] with B = -J, we conclude that for any 

7 &{J2a x x\a G (0, 1)1^1, a T l|f,| = l}, (6) 

there exists fi G (0, oo) d so that the mean stationary queue 
lengths are given by 71, ...,7rf. 

3) CSMA network on a partite graph: Consider the fol- 
lowing stylized model for a CSMA network [24]. Suppose 
there are n k class- A; nodes, with k = 1,...,K. If a node 
is active (transmitting) of say class-c, all nodes of classes 
fc ^ c cannot activate (begin transmitting). Class-c nodes, 
however, can activate. Nodes of class-fc, k = 1,...,K, try 
to activate after an exponentially distributed time with mean 
1/vk- After successfully activating, a node deactivates after 
an exponentially distributed time with mean 1. 

We will keep track of the number of active nodes and 
which class they belong to. Specifically, let (k,l) denote the 
state in which I class-fc nodes are active, I G {0,1, ...,n&} 
and k G {1, ...,K}. The equilibrium distribution of this 
Markov process is given by ir^j) = Z~ l \i) v k- 

Now consider the following two control schemes: (i) the 
operator can choose any v G (0, 00) and set v k = v 
for fc = 1 , . . . , K , and (ii) the operator can set any v S 
(0, oS) K . Intuitively, the operator has less control over the 
network with scheme (i) than with scheme (ii). In practice, 
however, there could be compelling reasons to use scheme 
(i) instead of scheme (ii), such as reduced complexity and/or 
lower operation costs. In such situations, it is worthwhile to 
examine and compare the achievable regions of all available 
schemes. 

For scheme (i), we identify r = In v and write 

^,0W = ^yexp(2r + ln(7)) (7) 

which is equivalent to (Q~|) when A( k ,i) — I and b( k ,i) 
! n The right-hand side of (O can then be interpreted as 
the steady-state average number of active nodes. Using The- 
oremQ] we conclude that for every 7 G (0, maxfc = i ...^ rife), 
there exists a v E (0, 00) such that Ylk=i S"=i ^(M) = 7* 
In other words, by using control scheme (i) and setting v 
appropriately, the average number of active nodes can be 
made to equal anything between and maxfci vi .^nt. 



In the case of scheme (ii), identify ri = \o-Vi for i = 
1, K, so that 

WO = ^exp(Zr fc + In («*)). (8) 

Again if we compare to (Q}, we see that A(k .i = ll[k = i] 
and b(k.i) = m (" fc )- Furthermore, (O is to be interpreted 
as the steady-state average number of active class-z nodes, 
where i = 1, ...,fc. Using Theorem Q] we conclude that for 
every 

K nt 

1 € {J2Y,te(i,l)ei\<x G (0,l) |O| ,a T l| O | = 1} 

i=l 1=1 
K 

= {Y / Pkn k e k \f3e(0,l) K ,(3 T l K < l}, (9) 

fc=i 

there exists a v £ (0, oo) A ' such that Yli=i ^(i.i) = 7i 
for i = 1,...,K. We see that by choosing v appropriately, 
the average number of active nodes of every class can be 
controlled. As expected, more control can be exerted on the 
network with scheme (ii) than with scheme (i). 

The achievable region given in (O can intuitively be 
understood as follows. When precisely one element of v 
becomes extremely large, say v k — > 00, all class-fc nodes are 
active almost always and the average number of active nodes 
would be nfcefe. If two or more elements become extremely 
large, say v kl , v k2 — >■ 00, a large number of class-fci and 
class-fc2 nodes would be active for a large fraction of time 
(but never simultaneously). The average number of active 
nodes is then a weighted average of n kx e kx and n^e^. 




Fig. 1 . Process describing a CSMA network on a partite graph with K = 3 
classes and n = (2, 5, 3) T . 

D. Proof of Theorem [7J 

In this section, we prove Theorem Q] Our method is based 
on [9], [10], where the achievable region of the throughput 
of nodes (transmitter-receiver pairs) in a CSMA/CA network 
has been determined. We apply the approach to the broader 
class of product-form networks and provide all necessary 
adaptations, which leads to Theorem Q] A proof sketch is as 
follows. First, we construct a convex minimization problem 
in r for every vector a G (0, such that l T a = 1. 
This minimization problem is constructed such that in the 
minimum located at r opt , A T 7r(r opt ) = A T a. Next, we show 
that strong duality holds and that the dual problem of our 



minimization problem attains its optimal value. This then 
implies that the target 7 = A 1 a is achievable. 

We now proceed with the proof. Let In a; = 
(lna;i, ...,lnx n ) T for x G R™ and define the log-likelihood 
function u(r) = — a T ln7r(r), with a G (0,1) ' n ' and 
l T ot = 1. After substituting (fl]), we find that 

u(r) = In Z(r)-a T (Ar + b). (10) 

Let g(r) = ~V r u(r) with V r = (d/drx, d/dr^) 1 - , so that 

g (r) = A T Tv(r) - A T a. (11) 

The log-likelihood function in ( TTOb has the aforementioned 
properties that we are interested in. 

Proposition 1. The function u(r) = — a T \mr(r) has the 
properties that (i) inf rgR d u(r) > 0, (ii) u(r) is continuous 
in r, (iii) u(r) is convex in r and (iv) at the critical point 
r opt for which g(r opt ) = 0, A T Tv(r opt ) = A T a. 

Proof, (i) By irreducibility, we have that ir x > for all 
x G O. Because a x > for all x G O, inf reR d u(r) — 
inf rgR d — a T ln7r(r) > 0. (ii) Being a composition of 
continuous functions in r, u(r) is continuous in r. (iii) Being 
a composition of a convex log-sum-exp function v(s) — 
\n(J2xen ex P Sx ) ~' yTs an d an affine transformation Ar+b, 
u(r) = v(Ar + b) is convex in r [4]. (iv) This follows after 
equating (fTTT i to 0. □ 

We next prove that there exists a finite r° pt for which 
g(r° pt ) = 0. Using Proposition Q] we conclude that if such 
a vector exists, then A T 7r(r opt ) = A T ot. In other words, the 
target vector A T a is attained by setting r = r opt . 

Consider the following optimization problem, which we 
call the primal problem. 



maximize — /3 T In (3 + ((3 — ot) T b, 

over /3e(0,l)l°l, 
subject to A T f3 = A T a, 1 T (3 = 1. 



(12) 



It has been constructed in such a way that its dual is the 
minimization of u(r) over r. This is by design, and we will 
use and prove this throughout the remainder of this section. 
We note first that (fT2l differs from the minimization problem 
considered in [9], [10], in that there is a second term {(3 — 
a) T 6 necessary to capture the broader class of product-form 
networks and to allow for the selection of parameters. 

Before we can prove that the minimization of u(r) over 
r is indeed the dual to (fT2b . we need to verify that strong 
duality holds. 

Lemma 1. Strong duality holds. 

Proof. Slater's condition [4] tells us that strong duality holds 
if there exists a (3 such that all constraints hold. Considering 
(3 = a. completes the proof. □ 

Strong duality implies that the optimality gap is zero, 
specifically implying that if (TTZb has a finite optimum, its 
dual also attains a finite optimum. 



Proposition 2. The optimal value of the dual problem is 
attained. 

Proof. Assume that the optimal solution x of ( TT~2T > is such 
that Xx = for all x in X. Being the optimal solution, x must 
be feasible. Recall that a is feasible by assumption. Any 
distribution on the line that connects a. and x is therefore 
also feasible. When moving from x towards a, thus along 
the direction ot — x, the change of the objective function in 
( [T2l i is proportional to 



(a - X ) T V^(-/3 T ln/3 + ((3 - cx) T b)\ a= 
<a-x) T (-lnX-l + b). 



(13) 



For x ^ I, Xx > so that — In \x — 1 + b x is finite. For 
x G I, Xx > so that (fT3l equals 00 (recall that a x > for 
all x G £1). This means that the objective function increases 
when moving (3 away from x towards a. It is therefore 
not possible that x is the optimal solution, contradicting our 
assumption. 

The optimal solution must be such that (3 X > for all 
x G O. This implies that — (3 T In (3 + ((3 — a) T b < 00 and 
using Slater's condition [4], we find that the optimal value 
of the dual problem is attained. □ 

Strong duality also implies that there exist finite dual 
variables so that the Lagrangian is maximized. These dual 
variables turn out to be precisely r opt . So what remains is to 
show that the dual problem to ( Tl2b is indeed the minimization 
of u(r) over r and that the finite dual variables for which 
the Lagrangian is maximized are indeed r opt . 

Proposition 3. The dual problem to (fTZt is 



minimize 
over 



u{r), 



r G 



(14) 



Proof. By strong duality, we know that there exist finite dual 
variables r opt G M d , w opt G [0, oo)l°l and z opt G R such that 
the Lagrangian 

L(f3; r opt , ™ opt , z opt ) = -0 1 \nf3 + ((3- afb 

+ (A T f3 - A T c*) T r opt + f3 T w opt + (l T /3 - l)^ opt (15) 

is maximized by the optimal solution /3 opt . By comple- 
mentary slackness, w° pt = 0. Because /3 opt maximizes the 
Lagrangian, 

V p L{f3 opt \ r op \ w op \ z opt ) = - In f3 opt - 1 (16) 

+ b + Ar opt + z opt l = 0. 

This equation can be solved for /3 opt , resulting in /3 0pt = 
exp ((z opt - 1)1 + Ar opt + b). The constant z opt follows 
from the normalizing condition l T /3 = 1, implying that 



13' 



;Opt 



exp (Ar opt + b) 



Z(r° pt ) 

where Z(r opt ) = 1 T exp (Ar opt + b). 



(17) 



Because /3° p is the optimal solution, we have that 

max L(P; r opt , w op \ z opt ) = i(/3 opt ; r opt , w opt , z opt ) 
/3e(o,i) |S2 

= -/3 optT ln/3 opt + (/3 opt - afb + (A T /3 opt - A T c*) T r opt 

= -/3 optT (ln/3 opt - Ar opt - 6) - a T (Ar opt + 6) 

= lnZ(r opt ) - a T (Ar opt + b) = u{r opt ). (18) 

Because /3 opt and r opt solve the primal problem ( fT2] >. r opt is 
the solution of min^Rd u(r). □ 

This concludes the proof of Theorem Q] 

III. Algorithm 

Having identified the achievable region in §0U we now 
turn to developing an algorithm that finds r opt . For this, 
we modify a (distributed) online algorithm developed and 
discussed in [22]. 

A. Description 

We apply the algorithm to the objective function u(r) = 
— a T ln7r(r), where a 6 (0,1)'°' is such that l T a = 1 
and A T ct = 7. When 7 is achievable, we know that such 
a distribution exists by Theorem Q] From Proposition [1] 
we see that u(r) is convex in r and that the gradient 
V r w(r) = ^4 T 7r(r) — 7. In its unique critical point r opt 
where V r u(r opt ) = 0, we indeed have that ^4 T 7r(r opt ) = 7. 
The algorithm will find this critical point. 

The algorithm is as follows. Let = *M < < ... and 
take initial parameters r = Ji' '. At time t^ n+1 \ marking the 
end of observation period n + 1, calculate 

nL" +1] = "TTTTi TT / t[Z [n \t) = a;]df (19) 

£[n+l] _ tl"i J t[n] 1 W 

for every state a; € f2. Here, {^^(i)}tN<t<t[«+i] is a time- 
homogeneous Markov process, which starts in Z^' 1 ] 
and evolves according to the generator of {X(t)} t > that 
corresponds to parameters R} n ' . Then update all parameters 
by setting 

R [n+1] = [R [n] _ a [n+l] n [ " +1] _ 7 )]« ( 20 ) 

Here, a[™ +1 ' denotes the step size and the truncation [r] TC is 
component-wise defined as 

[r]f = max{K lin , min{^r x , n}} (21) 

for K = [nY™,^ 3 *] x ... x [ft' d nil \7^ ax ], which denotes 
the set of available parameter values. 

Conditions on oW and /["I = l/(iM -^-i]) that ensure 
convergence of (1201 can be found in [22], but only when 

< 7^ ax - VJ2 m < 00 for i = l,...,d. Determining 
the achievable region for this set of available parameters 
requires solving a potentially NP-hard inversion problem 
[18]. Instead, we would like to use Theorem[T] but TheoremQ] 
only holds under the assumption that [K™ in , 7£™ ax ] = R for 

1 = 1, d. 

We therefore need to establish new conditions on and 
that ensure convergence for the case that TZ = R d . Such 
conditions can be found by modifying the proof in [22, §IV], 
which is the topic of the remainder of this section. 



B. Conditions for convergence 
When K = R d , we write 

R [n+l] = R [n] _ a ln+l] {A Tll ln+1] - j). (22) 

For the algorithm in d22l >, we will prove following result. 

Theorem 2. The sequence W n ' generated by the online 
algorithm (1221 l converges to the optimal solution r° pt with 
probability one, if a^ n \ and fl n ] are such that 

(ii) E^a [n KE™=iaH)(eW + exp(c 2 El =1 « W - 
c 3 (e /W ) exp(-e 4 Em=i a[m1 ))) < oo for all c% c 3 , 
C4 6 (0, oo). 

In Proposition |U we give two sets of sequences a^, 
and fl n ], labelled (a) and (b), such that conditions (i) and 
(ii) in Theorem [2] hold. The proofs that these sequences 
indeed satisfy the conditions are deferred to Appendix [A] 
The choices made for the step sizes and observation periods 
in (a) are based on similar ideas as in [9], [10]. Note however 
that conditions (i) and (ii) differ from the conditions in [9], 
[10], and the verification that (i) and (ii) hold is therefore 
different. 

Proposition 4. Conditions (i) and (ii) in Theoretn\2\hold for 

(a) aW = (nln(n + e>J = n~ a / 2 and /M = n~ 5 
where a > and 8 > 1 + a, and 

(b) aW = n~\ eW = (n / 2 ^^ 1 ) -1 and f [n] = 
(Inn + l)~ 2 n~ s where a > and S>l + a + C4 — 
l + a + c g (l +2max !/e f2 

C. Convergence proof 

We start by defining the error bias = 
E[G [ "V [ ™~ 1] ] - G [n] and zero-mean noise = 

G [n] - E[G [ ™ 1 |J" [ ™^ 11 ]' respectively. Here, T [n - 1] 
denotes the cr-field generated by the random vectors 
Z^ZW where Z® = (R [a] ,X(0)) T and 

z [n] = (G W ,H["],X(t["]))T f or n > 1. We will proceed 
to use Lemma 12 proven in [22, §4.1] 

Lemma 2. The sequence i?'™' generated by either on- 
line algorithm (120b or (122b converges to the optimal 
solution r opt with probability one, if the summation 
J2™ =1 al n ]\E[B [n]T (R [n ~ 1] - r opt )\^ nLl ]]\ is finite with 
probability one and if also, for any e > 0, there exists 
mo 6 N so that for any n > m > m , 

(i) E"= m aWB [3]T (r° pt - R [] ~ 1] ) < e and 

(ii) E" =m a^E^ T {r opt - R^-^) < e 
with probability one. 

Proving Theorem [2] thus boils down to verifying the 
conditions in Lemma [2] for d22l >. This is however more 
complicated than for d20l i. which has been done in [22, 
§IV-B]. This is because it is possible for a component of 
run off to ±oo. Fortunately, because || V r u(r) ||2 < 
c g = |fi|c?max Xi i \A x> i\ [22, §III-B], the rate at which the 
algorithm can be bounded. 



Lemma 3. For n 6 N and i = |i?[ nl | < \R [ ° ] \ + 

spn [ m ] 

Proof. Fix ncN and use the triangle inequality repeatedly 
to obtain 



\R 



\R 



[n-l] 



<N 01 i + E 



(Mix , 



n 



This completes the proof. 



(23) 



□ 



We now turn to verifying Lemma [2] To do so, we consider 
the error bias and zero-mean noise separately, similar to [22, 
PV-B]. 

1 ) Error bias: Using Lemma [3] we find that 



°pt\|^7[n-l]i 



Y^a [n] \E[B [n]T (R [n ~ >l 

71=1 

oo d 

=E aW IE E N ,il i^ [ "" 11 ](^"" 1 

n— 1 i—1 

oo eZ 



Opt x 



n— 1 i—1 

oo ci 

<E aN EN nl Ki« 

n— 1 i—1 

oo d 



[n-l] I I opt I 



<E aW Ei B ! nl i(i^ 01 i + K pt i+c g E 

ri— 1 i—1 m—1 

oo d 

^{i^i + irrDE^'Ei^ 

n—1 i—1 

oo d n—1 

+^E( aW E^ nl iE« [ ' 



< 



(24) 



This expression is similar to [22, Eq. (36)], but applies to 
algorithm (l22l instead of algorithm (|20| |. Next, we turn to 
bounding from above. Expression [22, Eq. (37)] says 

\Bf ] \ < |E E 0nL n] -* x {R [n - l] )\\F [n - l] ] (25) 

and still holds in the present case, because \gi{n) —gi{u)\ < 
2max Xji {|A Cii |}||/2-iv|| var = c\\\n - i/|| var fori = l,...,d 
[22, §III-B]. When using algorithm d22l . however, we cannot 
proceed and apply [22, Lemma 7]. Instead, we will use 
Lemma |U the proof of which can be found in Appendix 
iBl We derive Lemma |4] using a large deviations result in 
[7], and we note that it is related to the mixing time of the 
underlying stochastic process, see also [16, Thm. 12.4]. 

Lemma 4. There exist c\, C2, C3, C4 £ (0, 00), so that for 
e N e [0, 1] and xeil, 



P[|n|™ ] - ir x (R [n - 1] )\ > e [ 

n 

< a exp (c 2 E ° [mI _ c 3 



(26) 



( P H\2 « 

i^fexpH.E^ 1 ))- 



Similar to [22, Eq. (38)], we conclude that 



10M1 ^ c il^l 
-£>] < — - — max 



{l,ci}(eW + 



(27) 



ex P (c 2 E « H - C3 ^f exp (-04 E « [m] ))) 

m=l ^ m=l 

Substituting d27b into d24l) . we conclude that 

00 

Ea W |E[-B [n]T (i« [n_11 -r- opt )|J-[ n - 1 ]]| < 00 (28) 



[n]\2 



n=l 



with probability one if aW, eW and /W are such that 



n-l 



E« w (E« H )(e 



(29) 



n—1 m—1 



exp 



( C2 E 



n]\2 



a L J - c 3 - 



( eW) 
/M 



exp 1 — C4 > a 



E 



< 00, 



which completes the proof of Theorem [2] 

IV. Conclusion 

We have identified the achievable region for Markov 
processes with product-form distributions when an operator 
can control one or more parameters (transition rates) of the 
system. The shape and size of the achievable region was 
shown to be intimately related with the state space and the 
parameters that can be controlled. Loosely speaking, the 
larger the state space is, the more configurable parameters 
there are, the more performance measures there are that can 
be achieved. 

Future work may be aimed at relating the achievable 
region to parameter domains. In the present work we as- 
sumed that parameters can be arbitrarily set, while in practice 
operators can only choose parameters from a compact set. 

We also described how to use a (distributed) online al- 
gorithm from [22] to find parameters such that the perfor- 
mance measure of the system equals any target performance 
measure taken from the achievable region. This required 
broadening the scope of applicability of the online algorithm 
in [22], because the shape of the achievable region is only 
known to us when transition rates are unbounded. By capi- 
talizing on and generalizing an existing proof methodology 
[22], we have provided sufficient conditions that guarantee 
convergence of the algorithm when an operator can arbi- 
trarily set parameters. Because parameter values were now 
assumed unbounded, the conditions on the step sizes o'™' 
and observation frequencies had to be more stringent in 
order to lessen the risk of extreme parameter growth. 

Further research on the algorithm may be aimed at study- 
ing the necessity of such stringent conditions. Less stringent 
conditions possibly allow for increased convergence speeds. 
To further substantiate this, a better understanding is required 
of the delicate interplay between the mixing times and the 
convergence properties of the algorithm. 
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Appendix 
A. Proof of Proposition 

First define Z,M = E" l= i( m ln (m + l))" 1 and ffH = 



ESUCmhm)- 1 = °° *nd E^i(« W ) 2 < E™ =1 < 
oo. Next, we examine condition (ii). Note that 



ra-l 



E- w (E« [mI > w ^E^r 



n=l 



m— 1 



n i+«/2 in (n + 1) 

n— 1 v 7 



(30) 



is finite, because fft™" 1 ] < In (n - 1) + 1 < In (n + 1) + 1 
for all n > 2 and a > 0. We still need to check whether 



n—l 



a [n] (E fl[ml ) ex p( c 2 E alm] - c 3 



n—l m— 1 



m— 1 



( e H)2 



n oo £[n-l] 

exp (-c 4 £ «W)) = J2 n]n(n+1) exp(c 2 i, 

m=l n—l ' 



c 3 n 5 - a exp (-c 4 i W )) 



(31) 



is finite for all C2, C3, c 4 G (0, 00). For this, we note that 

lN = J_ + y I < JL + y _J 

i2 ^— ' mln(m+ll In 2 ^— ' mk 



< 



In 2 ^— ' mln(m + l) In 2 ^— ' to In to 

m=2 v ' rn=2 

2,,2 J 2 ^- l ^ + 2^- 1 ^- (32) 



Define cl — 3/(2 ln2) — ln(ln2). Consider the summand of 
~~2r 1 an^u^pSf r febund it for all n > 3 by writing 



£[»-!] 

— — — — - exp(c 2 L„ - c 3 n s - a exp (-c 4 L W )) 
nhi (n + 1) 

ln(ln(n - 1)) + c L , S - a i r M\\ 

< . . — —. exp(c 2 L n - c 3 n° exp (-c 4 L ln| j) 



nln (n + 1) 



< 



1 + C L 



■ exp 



(c 2 i„-c 3 n A '- Q exp(- C4 i [ " 1 )). 



(33) 



We next upper bound the exponential. Because the exponen- 
tial function is monotonically increasing and c 2 , c 3 , c 4 > 0, 
we can upper bound again by 



< 



'(1 + cl) 



(lnn) C2 exp (-c 3 e- C4Cl n 5 - Q (lnn)- C4 ). 



Now note that for any e > 0, In n < n e for all n sufficiently 
large, implying that 



< 



•(l + c L ) 



(lnn) C2 exp (— c 3 e CiCL n 



CiCLr.S—a—ae 



) (34) 



for all n sufficiently large. Define e = S—a — c^e and choose 
e G (0, (6 — a — l)/c 4 ], so that e > 1. We have created 
a sequence that is an upper bound for OTT ) and converges 
absolutely, which can be seen using the ratio test, 



lim 

n— >oo 



' ln(n + 1) \ c 2 __n_ eC3C -c i c L ( „ e _ (n+1)e) 
In n ) + l 
if £ < 1, 







if e = 1, 
if e > 1. 



(35) 



En 
m=l ' 

Let aW, e N and /N be as in (a). Condition (i) is 
satisfied, because Y^=\ a '™' = E^Li( nm ( n + I)) -1 ^ 



Next, let aW, eW and /N be as in (b). Verification of 
condition (i) is immediate. To verify condition (ii), we first 



note that "£n=i ftN (Em=i a W ) e [,i] = £n=i n' 1 ^' 2 < Then note that 



oo. What remains is to check whether 

( e N)2 



minjexp {AR [n] + b) x \ = exp (min{(AR [n] + b) x \) 



E « N ( E ^ ex P( C2 £ aW " C3 -^FT > exp f min{6 x } + min{V A y z R [ ' l] }) > expfmin^} 

n=l m=l m=l J ^ xeU y eU ~[ 1 \x&1 

x exp (-c 4 £ a [m] )) = E «P(^ - max{£ 1^,11^1}) > exp(min{6 x } 

m=l n=l i/GS] ^-^^ ' J \i6S! 



m=l n— 1 y£f 

Tnn + 1\ 2 



-c 3 n s - a ( ,,, ) exp(-c 4 H^)) (36) r rnl " r \ 

Vifh-iiy PV " -.max^l^l+cg^a^n^ll^.lli). (40) 



is finite. For n > 2, the summand can be upper bounded by 



m— 1 

Here, max ye si denotes the maximum absolute row 



jj[n-i] /lnn+l\ 2 sum of A. Similar to d40li. max TP n{exp (AR [ni + b) B } can 

— - — exp(c 2 i? [,il - c 3"-' 5_a (-^— []-) ex P (- c 4-ff M )) be upper bounded, and one finds that 

< e c ^-- 1 (lnn+ 1) exp (-c^n 6 —^ ) . (37) W > 1 cxp ( min{M _ max{M 

After defining g = J — a — ca > 1, we conclude using the r , r n i, r™ii \ 

ratio test that " 2 , J&ti^l + * E ^ > *§? " Ar.-lli) ■ («) 

m— 1 

ln(rt + 1) r n + 1 \ C2-1 C3C -'4( n '_( 7I+1 )S) Xo summarize, there exist c\ = exp(ma,x x& n{b x } 

n v n ) 



lim 

n— >oo 



In 



min 2:6 n{^} + 2max J ; = i : ... :ti {|i?f 1 |}max ye o ll-Ay.-Hi) > 

1 if e < 1, and c 2 = c g max ye n > so that 

e- c * e ~ CACL ife=l, (38) 

if e > 1. ^mL)"' <c lCX p( C2 E« [m1 )- ( 42 ) 

m— 1 

This proves absolute convergence, since s > 1. □ 2) Bounding k: From the definition of k, it follows that 

K <E E llTar.J/IU = E E h*,v\\<l> 

B. Proof of Lemma^ eeE { 7x , y er|eG7 a;i! ,} x,j/ef2 e67x,„ 

Consider the following setup, similar to that in [8] for = E b^slhx.ylU < \G\ 2 \E\ max {HTx.J^}, (43) 

discrete-time Markov chains. Define a graph G — (V, E), x, v en 

where V denotes the vertex set in which each vertex cor- and from the definition of the path length it follows that 

responds to a state in ft and E denotes the set of directed ^ ( 1 1 

edges. An edge (x, y) is in E if and only if 0(e) = "K x Qx,y = \\lx,y\\<j>= E — n — - ~M~ ma n 1 7> — I' 

T^yQy,x > 0. Here, Q denotes the generator matrix of e=(z,v) T <E~( Xty ' "min ^ ' 

{X(t)}t>o- For every pair of distinct vertices x,y G fi, r n i 

choose a path lx , u (along the edges of G) from x to y. Rec ^ that R i corresponds to the logarithm of a rate, 

Paths may have repeated vertices but a given edge appears i- e - = m Qx,y for some x, y G fi. It follows that for 

at most once in a given path. Let V denote the collection of C Q = max { z ,ven\^ ie{1 d yR [ * li =inQ z>v }{Qz,v} < °°> 

all paths (one for each ordered pair x, y). Irreducibility of ^ ^ 

{X(t)}t>o guarantees that such paths exists. For ^ x y e V max I — — > < max {e~ Ri } + cq < max {e'^ '} 

define the path length by h^.H^Ee^^^e)" 1 : Also, *'» elU J i = 1 -" 1 ^ ^ 

let « = max e E {7x , H er|ee 7 ,. H } Il7*,» U^y + cq < . max {e'"* l}e c ^-=i Q +c Q . (44) 

With this set-up, the upper bound P[|flL™ ] -^(.R^ 11 )! > 2 2 2 fm] 

e Wl <f>] )-| exD f_f e N)V4 K lfi 2 fN) W here7r [nl - Using© k < c\\G\ 2 E\ 2 exp (2c 2 £ m=1 a M ) x 

e J ^ l^minJ 2 ex P ( l e L J /4K» / LJ ), wnere it ■ - . .. p i U „ Ce v« , a M , „ \ - ji^id^ v 

min^nTT,^" 1 ), has been derived in [22, Appendix D]. m3X f gi f« D> e + ^ ^ £ G l . X 

We will show that LemmaHcan be derived using this upper (max*{ei , } + CQ ) exp (( C g + 2c^ ^ m=1 a' or equiv- 

bound, by calculating a lower bound on tt)™^ and an upper alentl y K - exp ( C4 £m=i a ™ )' where 

bound on k when using the online algorithm in (l22l . 2 / r \ _1 

7) Bounding 7r^ n : By non-negativity of exp (•), C3 = |fi|c 2 |G| 2 |i;| 2 L iwl 6 J + CQ J > ° (45) 

w and c 4 = c g + 2c 2 = c g (l + 2 max^n > 0. Noting 

n H > mm x gn{exp(Ail^ +b) a ) ( ^ that | G | = |^| < oo and \E\ < - l)/2 < oo 

mm |f2| max^gnjexp (AR^ + b) x } completes the proof. □ 



